Gesture enabled keyboard

ABSTRACT

A gesture-enabled keyboard and method are defined. The gesture-enabled keyboard includes a keyboard housing including one or more keyboard keys for typing and a pair of stereo camera sensors mounted within the keyboard housing, a field of view of the pair of stereo camera sensors projecting substantially perpendicularly to the plane of the keyboard housing. A background of the field of view is updated when one or more alternative input devices are in use. A gesture region including a plurality of interaction zones and a virtual membrane defining a region of transition from one of the plurality of interaction zones to another of the plurality of interaction zones is defined within the field of view of the pair of stereo camera sensors. Gesture interaction is enabled when one or more gesture objects are positioned within the gesture region, and when one or more alternative input devices are not in use.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/471,494, filed May 15, 2012 to El Dokor, titled Gesture KeyboardMethod and Apparatus, currently pending, which in turn claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/619,915 filedApr. 3, 2012 to El Dokor, titled “Gesture Keyboard Method andApparatus”, the contents of these applications being incorporated hereinby reference.

FIELD OF THE INVENTION

This invention relates generally to a computer or other keyboardsupporting the use of gesture commands, and more particularly to acomputer keyboard including one or more cameras therein to image a usergesture, and to allow for the processing of this imaged user gesture toissue one or more commands to a processing system.

BACKGROUND OF THE INVENTION

As Kinect®-like devices become more prevalent in the consumer space,they are empowering a brand new generation of user interfaces, ones thatare enabled with gesture recognition. Front-facing gesture recognition,with cameras placed in peripheral devices for game consoles or PCs, isvery powerful, enabling fun, aerobic gesture controls for games. Theinventors of the present invention have recognized that this approach,however, suffers from a number of drawbacks: often times, sensors don'twork very well, or even at all, in regions that are close to the screenor lower in the field-of-view (FOV) of the sensors. Because these areessential areas of interaction for the user, these users are left withthe choice of foregoing such interaction, or attempting the interactionat great discomfort. With cameras placed in the bezel of a screen, someregions of desired interaction cannot be viewed by the camera, even ifthe cameras are provided having a wide field of view. More importantly,for current technologies being utilized in gesture recognition witheither time-of-flight (TOF) or structured light, peripherally locatedobjects relative to the FOV may not be easily viewed by the camera, andmay have poor reflectivity, vis-à-vis a light source that is associatedwith camera or image sensor. Many such objects may have slanted surfacesat such locations in the FOV that image distortions, associated withboth the lenses and the light source, present a very serious challenge,both, in engineering and usability.

Many are the advantages of gesture recognition, enabling a more naturalmeans of interaction between the user and their surroundings. Gesturerecognition today is especially attractive for playing games, byenabling user gestures. This is evidenced by millions of Kinect® userswho have taken to such interfaces very naturally. The gesture lexiconthat is associated with a gesture recognition system is typicallysimplistic, involving coarse body gestures and tracking the user'sentire body in the FOV, which is impractical for near-touch interaction.In this case, the gesture lexicon itself is monolithic, mostly recyclingvery similar gestures across multiple games.

While gesture recognition has proven to be very valuable for games, ithas been considered less valuable in other settings that require moreprecise motion tracking, or that require a greater degree of robustness,more integrated interaction with other types of controls, and asignificantly more diverse gesture lexicon. For such applications, theidea of using such coarse gestures becomes less appealing. Moreover,user/arm fatigue, among other factors typically becomes more of anissue. It would be desirable to develop another approach for moreuser-centric gesture recognition, capable of addressing the shortcomingsof the current state-of-the-art.

It would therefore be beneficial to present a method and apparatus forovercoming the drawbacks of the prior art.

SUMMARY OF THE INVENTION

Therefore, in accordance with one or more embodiments of the presentinvention, a new approach is presented that addresses the shortcomingsof the prior art, and that integrates user-centric gesture recognition(vs. screen-centric) with conventional mouse/keyboard interfaces. Inaccordance with one or more embodiments of the present invention, a newway of approaching the integration of gesture recognition into moretraditional controls is presented; through which users can have a morenatural computing environment. Sensors are preferably integrated intothe keyboard with lenses enabling a wide field-of-view. Once a user'shands are lifted about four inches or so (or other desired distance)above the keyboard, the hands appear as objects in the field-of-view.The gesture recognition portion of the interface can then be activated,and users can interface with whatever user interface they wish. Userscan also lift a single hand or both hands above the keyboard. Thus, theinventive system provides for a delineation of gesture recognitionbetween an active zone that is enabled once the user's hand (or hands)is visible for the cameras and inactive zone when the user is typing onthe keyboard or using the mouse. In such a situation, not only are theuser's hands not visible by the cameras, but the use of the keyboard ormouse may be employed to further differentiate when the user wishes toactivate the gesture recognition capabilities of the system. Of course,any combination of hand placement, mouse, and keyboard operation may beemployed to determine activation of the gesture features of the system.Through the use of these various user interactions, a powerfulinteraction model can be provided to users and thus allowing for thecomfortable use of gesture interaction in a system in which it wouldpreviously have been considered less useful.

In accordance with one or more embodiments of the present invention, thethree-dimensional space above the keyboard may be partitioned intovarious interaction zones. Thus, a first zone may be provided above thehands of the users in which a first gesture may be recognized, such as astatic or other gesture. A second zone may be provided adjacent thefirst zone so that movement from one zone to the other can bedetermined, thus allowing for the determination of a dynamic gesture,for example. Any number of such zones may also be provided, thusallowing for a determination of movement in any number of directions ordistances. In accordance with these various embodiments of the presentinvention, users may be encouraged to use gestures for small,time-saving tasks, and not struggle with complex and tedious gestures,requiring a significant amount of coarse arm movements, although thesystem presented herein would be equally applicable to such gestures. Asa result of the aforementioned information, there exists a dichotomy inwhich gestures enable fast-access tasks, such as minimize, maximize,pinch, select, etc. The more tedious and time-consuming tasks may stillbe conducted with more conventional interfaces such as a mouse andkeyboard if desired by the user.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specifications anddrawings.

The invention accordingly comprises the several steps and the relationof one or more of such steps with respect to each of the others, and theapparatus embodying features of construction, combinations of elementsand arrangement of parts that are adapted to affect such steps, all asexemplified in the following detailed disclosure, and the scope of theinvention will be indicated in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 depicts a view of a keyboard including sensors and interactionzones in accordance with a first embodiment of the present invention;

FIG. 2 depicts a perspective representation of an active gesture region;

FIGS. 3(a) and 3(b) are flowchart diagrams depicting steps for gesturerecognition in accordance with embodiments of the present invention;

FIG. 4 is a flowchart diagrams depicting steps for gesture recognitionin accordance with an embodiment of the present invention;

FIG. 5 depicts sensor structure in accordance with an embodiment of theinvention;

FIG. 6 depicts sensor structure employing one or more MEMS in accordancewith an alternative embodiment of the invention;

FIG. 7 depicts the use of a single sensor element in accordance with anembodiment of the invention; and

FIG. 8 depicts depth estimation and scaling techniques in accordancewith an embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

One or more embodiments of the invention will now be described, makingreference to the following drawings in which like reference numbersindicate like structure between the drawings.

In accordance with the various embodiments of the invention, three mainapproaches to the integration of an inventive interaction dichotomyincorporating both traditional as well as multi-touch gesture events arepreferably presented. These different approaches may be characterized bythe following: the nature of interactivity that is associated with each;whether each involves the partitioning of the 3D space into active andpassive regions, and whether each involves static and/or dynamicgestures.

First Approach: Integration of Confusion Learning with Static Gestures

A first approach to the development of a gesture lexicon or having asystem learn a new gesture dictionary is preferably implemented throughthe utilization of confusion learning, defined in U.S. patentapplication Ser. No. 13/221,904, filed 31 Aug. 2011 to El Dokor et al.titled “Method and Apparatus for Confusion Learning”, the contentsthereof being incorporated herein by reference. As is described in thisincorporated application, the system is trained on various hand gesturesusing a Belief Propagation Artificial Intelligence system, and one ormore specialist subnets may be spawned to specialize in specificallyidentifying and/or predicting confusing user gestures. Once the gesturedictionary is trained, a useful approach would then be to integrate sucha dictionary with a user interface, such as the gesture recognitionsystem and apparatus as described in accordance with the variousembodiments of the present invention.

Second Approach: 3D Space Partitioning into Active “Touch Point” Regionsand Passive Regions

A second approach in accordance with an embodiment of the inventionpreferably comprises creating a thin three-dimensional virtual membraneat a predefined position between the keyboard and the screen(s). Theuser can then traverse the membrane at multiple locations, enablingmultiple touch points. This is preferably performed by mapping the localmaxima (spatial maxima) in the membrane one-to-one onto screen points,and then enabling gesture events at these points. This three-dimensionalmapping into two-dimensional touch points is efficient and intuitive,minimizing the learning curve.

Third Approach: Integration of Confusion Learning with Static GesturesAlong with 3D Space Partitioning

This preferred third approach consists of combining the first twoapproaches into a hybridized model that combines the intuitive benefitsof three-dimensional mapping with the efficiency of static gesturesthrough confusion learning. This approach is capable of addingsignificant value to three-dimensional gestures. Not only canthree-dimensional gestures emulate their two-dimensional counterparts,but these three-dimensional gestures can also expand on thefunctionality of the two-dimensional counterparts, adding new featuresand capabilities. From an object-oriented viewpoint, three-dimensionalgestures may be provided as one or more child classes of a parenttwo-dimensional class, inheriting various feature of the parenttouch-based class, while expanding on such features into new featuresthat may enhance user interactivity.

Overall Approach

In accordance with a preferred embodiment of the invention, andreferring first to FIG. 1, a keyboard 100 is shown including a pair ofstereo camera sensors 110 mounted on keyboard 100. These stereo cameras110 further define a primary interaction zone 120, and a secondary touchzone 130. When employing this inventive keyboard in accordance with oneor more embodiments of the present invention, depth is preferably firstreconstructed from the pair of stereo sensors 110 integrated intokeyboard 100, or otherwise associated with a keyboard or other “desktop”hardware component. As is further shown in FIG. 2 in conjunction withFIG. 1, the pair of stereo sensors preferably defines each of theprimary interaction zone 120 and secondary touch zone 130 in accordancewith a corresponding gesture region 200, having a height 210, depth 215and width 220 defined in accordance with the field of view provided bythe noted pair of stereo sensors. While gesture region 200 is shown as arectangular prism, any desired shape may be employed, with appropriatesensors being employed.

As is shown in FIG. 2, when an object, such as the finger 230 of a userenters into gesture region 200 (corresponding to either primaryinteraction zone 120 or secondary touch zone 130) the pair of sensorsrecognizes entry into the region, and then enables gesture control ofthe apparatus associated with the keyboard, such as a computer or thelike. It is preferable to define gesture region 200 as beginning apredetermined distance above the sensors and keyboard 100 so that thehands of the user do not enter gesture region 200 during normal typing,mouse use, track pad use or the like. Rather, upon actively raising oneor both of one's hands a predetermined amount, gesture interaction isenabled. Such predetermined height may be about four inches in anexemplary embodiment, but any desired height may be selected.Additionally, while gesture region 200 may be predefined, a user mayalso be permitted to determine the size and location of such gestureregion. Further, the definition of the size and location of the gestureregion may also be defined by learning of the habits of the particularuser.

Additional trigger events may further be defined to enable and disablegesture recognition in accordance with one or more embodiments of thepresent invention. Thus, one or more embodiments of the presentinvention provide for activation of gesture recognition that is enabledzone once the user's hand (or hands) is visible for the cameras andtherefore in an active zone, and that is disabled when the user istyping on the keyboard or using the mouse, and is thus their hands areout of a predefined portion of the field of view of the camera sensors,and thus in an inactive zone. In such a situation, not only may thelocation or position of the hands of the user be employed to enable anddisable gesture based upon the hands visibility to the cameras, but theuse of the keyboard or mouse may also be employed to furtherdifferentiate when the user wishes to activate the gesture recognitioncapabilities of the system. Any combination of hand placement, mouse,and keyboard operation may be employed to determine activation of thegesture features of the system. For example, when a user is typing onthe keyboard, gesture recognition may be disabled. When the user ismanipulating the mouse gesture recognition may be disabled. When theuser is typing on the keyboard with one hand, gesture recognition may beenabled for the other hand. When the user is manipulating the mouse withone (preferably predefined) hand, gesture recognition may be enabled forthe other hand. Through the use of these various user interactions, apowerful interaction model can be provided to users and thus allowingfor the comfortable use of gesture interaction in a system in which itwould previously have been considered less useful.

For a detailed overview of an example of a preferred depth approach thatmay be employed in accordance with the invention, as well as how theycompare to some of the other existing methods, the reader is referred toU.S. patent application Ser. No. 13/025,038, titled “Method andApparatus for Performing Segmentation of an Image”, filed Feb. 10, 2011to El Dokor et al., U.S. patent application Ser. No. 13/025,055, titled“Method and Apparatus for Disparity Computation in Stereo Images, filedFebruary 10 to El Dokor et al., U.S. patent application Ser. No.13/025,070, titled “Method and Apparatus for Determining Disparity ofTexture”, filed Feb. 10, 2011 to El Dokor et al., U.S. patentapplication Ser. No. 13/294,481, titled “Method and Apparatus forEnhanced Stereo Vision”, filed 11 Nov. 2011 to El Dokor et al., U.S.patent application Ser. No. 13/297,029, titled “Method and Apparatus forFast Computational Stereo” filed 15 Nov. 2011 to Cluster et al., U.S.patent application Ser. No. 13/297,114, titled “Method and Apparatus forFast Computational Stereo”, filed 15 Nov. 2011 to Cluster et al., andU.S. patent application Ser. No. 13/316,606, titled “Method andApparatus for Enhanced Stereo Vision”, filed 12 Dec. 2011 to El Dokor etal., the contents of each of these application being incorporated hereinby reference.

Referring next to FIG. 3(a) a logical flow associated with an embodimentof the invention in which depth is reconstructed from the sensors(cameras) included in the keyboard is shown. As is first shown in FIG.3(a), one or more images may be acquired in accordance with one or moreimages acquired by the pair of stereo cameras in the keyboard at step310. Next at step 315 the system preferably identifies the presence of auser's hands in the FOV from the acquired one or more images.Additionally, other intended preconditions for enabling gesturerecognition may also be determined. At step 320, depth is computed inaccordance with the pair of stereo cameras in the keyboard, and furtherin accordance with one or more of the techniques noted above (or anyother appropriate depth calculating method). Once depth is computed, theuser's hands are preferably tracked in step 325, and touch points aredefined with static gestures at step 330, preferably enabling specificevents in accordance with one or more recognized static gestures.

Referring next to FIG. 3(b) a logical flow associated with an embodimentof the invention in which depth is reconstructed from the sensors(cameras) included in the keyboard is shown. As is first shown in FIG.3(b), one or more images may be acquired in accordance with one or moreimages acquired by the pair of stereo cameras in the keyboard at step350. Next at step 355 the system preferably identifies the presence of auser's hands in the FOV from the acquired one or more images.Additionally, other intended preconditions for enabling gesturerecognition may also be determined. At step 360, depth is computed inaccordance with the pair of stereo cameras in the keyboard, and furtherin accordance with one or more of the techniques noted above (or anyother appropriate depth calculating method). Once depth is computed, theuser's hands are preferably tracked in step 365, and touch points aredefined with dynamic gestures at step 370.

Referring next to FIG. 4 a logical flow associated with an embodiment ofthe invention in which depth is reconstructed from the sensors includedin the keyboard is depicted. As is first shown in FIG. 4, one or moreimages may be acquired in accordance with one or more images acquired bythe pair of stereo cameras in the keyboard at step 410. Next at step 415the system preferably identifies the presence of a user's hands in theFOV from the acquired one or more images. Additionally, other intendedpreconditions for enabling gesture recognition may also be determined.At step 420, depth is computed in accordance with the pair of stereocameras in the keyboard, and further in accordance with one or more ofthe techniques noted above (or any other appropriate depth calculatingmethod). Once depth is computed, the user's hands are preferably trackedin step 425, and touch points are defined either with dynamic gesturesat step 430, or with static gestures at step 435, preferably enablingspecific events in accordance with one or more recognized staticgestures.

The different interaction paradigms represent different ways in whichsuch an implementation may take place in accordance with one or moreembodiments of the invention.

Defining the Mapping Functions of the Three Approaches

For the first approach (FIG. 3(a)), in which predefined gestures areintegrated into click and touch events, users' hands are preferablyfirst mapped onto the screen. Various touch gestures may be enabledthrough a gesture API or through sending click events.

For the second approach (FIG. 3(b)), two preferred mapping functions forthe above-defined two different interaction zones are preferablyapplied. The first is preferably a mapping function that is inherentlymany-to-one, mapping three-dimensional hand positions onto thetwo-dimensional scene. The second preferred mapping function definestouch events that are associated with various gestures. In accordancewith one or more embodiments of the present invention, thethree-dimensional space above the keyboard therefore may be partitionedinto various interaction zones. Thus, a first zone may be provided abovethe hands of the users in which a first gesture may be recognized, suchas a static or other gesture. A second zone may be provided adjacent thefirst zone so that movement from one zone to the other can bedetermined, thus allowing for the determination of a dynamic gesture,for example. In such a manner, crossing a border between suchinteraction zones may be employed to define a touch interaction or thelike. Any number of such zones may also be provided, thus allowing for adetermination of movement in any number of directions or distances.Mapping is preferably performed by first clustering changes in smallerregions near depth maxima, computing their centroids, and then mappingtheir values onto the second “touch zone” (See FIG. 1)

Consider the depth map, D(x,y,z), and a virtual touch membrane (orborder between defined interaction zones), defined at a specificy-coordinate location yi, T(x, y=yi, z). The binary mask associated withthe intersection of the two images defines the interaction region, andis preferably given by:I(x,y _(i) ,z)=D(x,y,z)∩T(x,y _(i) ,z)

For the third approach (FIG. 4), representing a combination of the firsttwo approaches, the two (or more) zones of interaction defined in thesecond approach are preferably be enabled, as well as some staticgestures, as defined in the first approach.

In an example implementation that typifies this approach, an email alertmay appear on the bottom of the screen as the user is typing and usingthe mouse. The user may extend his/her arm in the direction of thealert, maximizing the message. The user may then proceed to flippingthrough the email, minimizing it, expanding it, or simply closing it,all with hand flicks enabled with the proximity-based keyboard. The useris therefore able to continue to utilize gestures until he/she bringstheir hands back down to the keyboard again (or employs some otherdevice or the like that indicates a desire to disable gesturerecognition), at which point gesture recognition is suspended as theuser's hands leave the predefined interaction zone.

This approach represents a departure from frontal approaches that aretedious to enable in more natural settings where the user does notnecessarily wish to extend their arms too far away from their keyboardas they work. Computing requires ease of use, simplicity andresponsiveness, features that are easily enabled in accordance withembodiments of the present invention. Therefore, in accordance with thevarious embodiments of the invention, the user moves their hands orother gesture object in a direction that is perpendicular to the sensorlocations, rather than parallel to a field of view thereof. In thismanner, determination of movement between one or more definedinteraction zones is simplified, and thus allowing for the determinationof less exaggerated movements on the part of the user. In a stillfurther example in accordance with an alternative embodiment of theinvention, a user may play a virtual piano, the playing being preferablyimplemented with two interaction zones such that the primary zone isused to track the fingers of a user's hands, and the second zone istraversed by moving the hands forward. Movement from the primary zone tothe second zone (and thus traversing the border therebetween) preferablyindicates a touch interaction. The primary zone and touch zone arepreferably both three-dimensional interaction zones. Of course, anynumber of such interaction zones may be employed to provide stillfurther user control over desired interactions.

Hardware

As is next shown in FIG. 5, sensors 110 may be integrated into thekeyboard with only lenses visible at the surface of the keyboard 100.FIG. 5 depicts an overview of a novel apparatus combining the ability todetermine 2D and 3D gestures in a touchpad of a keyboard apparatus. Asis shown in FIG. 5, sensors 110 and lenses 520 are preferably mountedunder a glass pad 510. The keyboard's data communication protocol may bemodified to account for, both, streaming video as well as keyboardcommands. An alternative integration approach may also be enabled (shownin FIG. 6) by using micro-electronic machines (MEMs) 610, integratedunder the lenses 620, adaptively modifying the FOV as well as the focallength associated with the lens/cameras combination as desired orcontrolled in accordance with the MEMs 610, depicted in FIG. 6. FIG. 6depicts an overview of a novel apparatus combining 2D and 3D gestures ina touchpad with MEMs manipulating lenses. MEMs also have the ability toallow the sensors to follow the user's hands or other objects ofinterest in the field-of-view. These MEMs may move the entire baselineor just parts of it.

Alternative Hardware Configuration

A new approach to creating stereo images can be enabled by creating analternative hardware configuration in accordance with an alternativeembodiment of the invention that utilizes one or more mirrors to createa stereo effect. As is shown in FIG. 7, multiple views of a scene can bereceived at the same imager, on different parts of the imager, toproduce a stereo pair of images. Thus, scenes of an image 710 a, 710 bmay be received, and may be directed by mirrors 720 a, 721 a and 720 b,721 b, respectively to direct the image scenes to respective portions730 a, 730 b of a sensor 730. This may be accomplished by placing theimager (sensor 730) at the center of a harness (not shown), formaintaining and positioning multiple mirrors, thus leading to two imagesbeing subtended on the imager from two separate sets of mirrors,covering different angles of the field-of-view. Light is preferablyreflected from the scene onto the mirror configuration, and reflectedback onto the imager. The two images are therefore preferably receivedon the single imager and utilized in a similar set of algorithms thatwere presented earlier, and that have been presented in more detail inthe '038, '055 and '070 applications noted above. Adding MEMs, or arelevant switching mechanism coupled with one or more of mirrors 720 a,721 a, 720 b, 721 b, may enable the mirrors to be tilted inwards oroutwards, changing the baseline that is associated with the stereoimages, but also changing the point of convergence of the light.

Interaction—New Gesture Library

Given the interaction that is enabled through one or more devicesprovided in accordance with one or more embodiments of the invention,new interaction paradigms may be developed. One of which, mentionedearlier, may attempt to exploit existing touch screen gesture lexiconsto produce multitouch events via an existing multitouch lexicon. Onesuch approach defines two interaction zones, depicted in FIG. 1, inwhich one interaction zone defines the mapping region for the cursor andanother interaction zone defines the touch regions.

Combining Frontal and Keyboard-based Interactions

It may also be useful to combine the use of one or more peripheralgesture devices with the use of the inventive keyboard into the sameend-user applications. This is particularly useful if applicationsrequire multiple three-dimensional views. To enable such an approach,both input data streams from the two sensors may be integrated into thesame application. The data from the screen/monitor area may be generatedby either a time-of-flight or structure light sensor. It may also begenerated along the same approaches in the '038, '055 and '070applications noted above, or in any other known manner.

Updating the Background

A background model of the scene is preferably constantly updated toremove effects from lighting sources, peripheral motion, and any otheraspect of the field-of-view that may affect system performance. This newbackground can be constantly updated while, for example, the mouse orkeyboard is in use since the user's hands are not in the field of view.Once the mouse is inactive, or other indication of desire to initiategesture recognition in accordance with one or more embodiments of thepresent invention, gesture recognition is initiated and backgroundupdating is preferably suspended.

Enabling a pinch gesture: in one embodiment, a pinch gesture, verysimilar to the one that is available in multi-touch interfaces, isenabled by training a belief propagation AI on a number of input imagesrepresenting various stages associated with a pinch gesture. The pinchgesture can then be integrated into iOS and Windows 8 multitouch.

Refining the Depth Map

To provide more depth fidelity, an approach in which surfaces may befurther refined by segmenting them horizontally or vertically is alsodefined in the '038, '055, 070, '481, '029, '144 and '606 applicationsnoted above. Much along the same lines, depth estimates may be furtherrefined to compute more accurate depth estimates. As a result, accuratecomputation of the fingers three-dimensional location can be obtained.

Combining Touch and Gesture Keyboard

Another approach contemplated in accordance with various embodiments ofthe invention combines gestural interfaces with touch, to produce animplementation that enables users to use touch or multitouch. Users caneither touch the screen, or move their hands away from the screen,enabling gestures. In this approach multitouch gestures become a parentclass of 3D gestures, inheriting the parent gestures' many propertiesand states, but yet adding more features and more degrees of freedom tothese gestures.

Integrating Touch Screen with Dual Sensors

The dual sensors in this configuration can be placed underneathnon-scratch surface glass, with the lenses placed between the surfaceand the lenses as described above. A GPU is preferably used toaccelerate the computation of either 2D or 3D gestures, depending on aposition of the users' fingers that may be located either away from orclose to the glass. As a result, a control mechanism is highlighted inaccordance with an embodiment of the invention that combines 2D and 3Dgestures.

Integrating Frontal Controls—Exploiting Lens Warping

The peripherals of the field-of-view offer an opportunity to interactwith the system while users are standing up or are otherwise parallel tothe cameras or otherwise outside a center of the field of view of thecameras. Radial distortion is usually considered a lens/geometricaberration that is caused by the way lenses bend rays of light (Sonka,Hlavac, & Boyle, 2008). While radial distortion and its associated sideeffects are generally treated as problems requiring dewarping functionsand fixes, such aberrations may give valuable insight into depth-basedinformation. This is true even for peripherally located objects. Anidealized distortion is presented, where objects on the periphery givevaluable insight into the three-dimensional information of the lenses'FOV. Scenes with fisheye lenses or even some wide-angle lenses are notonly corrected for radial distortion, but in a stereo implementation,they also have to be corrected for in the z-dimension. This is necessarysince such lenses project 3D information onto the imager's x-ydimensions. Radial distortion functions that are typically associatedwith two-dimensional (x-y) values become inadequate in capturing trueevents being displayed in the field-of-view. A typical radial distortionfunction, r, is given by (Sonka, Hlavac, & Boyle, 2008)r=√{square root over ((x−x _(o))²+(y−y _(o))²)}where (x,y) represents a pixel in the image that is distortionless, and(x_(o),y_(o)) represents the principle point. The projection itselfcaptures parts/components of all three dimensions, and is inherent tothe distortion, in approximately radially increasing distances from theprinciple point. The projection also has the same dimensions as thereal-world data, making it, more specifically, a bijection. Depth mayalso preferably be reconstructed from stereo sensors in a manner similarto the '038, '055 and '070 applications, as mentioned earlier. One nowhas a radially distorted depth reconstruction (x,y,z), being projectedon (x′,y′), with z being approximated by the computed z′. Partialextraction of the true depth value can be accomplished, while evaluatinga transformed, distorted image.

An Alternate Approach to Stereo with Omnidirectional/Fisheye Lenses

Fisheye, or Wide-Angle Lenses offer significant advantages over narrowerfield of view lenses. In addition to providing a wider FOV, the degreeof distortion in the field-of-view can sometimes be indicative oflocation of objects, relative to the baseline that is associated withstereo sensors. However, most approaches to computational stereo withfisheye/omnidirectional lenses have mostly focused on de-warping theimages first. In doing so, images are fixed for their radial distortion,as mentioned earlier, and then a given disparity estimation approach isutilized. In accordance with one or more embodiments of the presentinvention, the problem of computing stereo from a pair of sensors withomnidirectional lenses is considered and implemented quite differentlyfrom standard methods.

Various relevant equations that are associated with the inventive methodfor computing depth, relating depth to disparity, as well as depth-basedscaling will now be presented.

Depth Estimation and Scaling Relationships for Computational Stereo

As illustrated in FIG. 8, depth scaling may be derived from therelationship between an object's actual size, a, and its apparent sizein pixels on a camera sensor, a′.

The relationship between the size of an object and its distance from thecamera is given as:

$\begin{matrix}{a^{\prime} = {a\left( \frac{fk}{D} \right)}} & {{Equation}\mspace{14mu}{A1}}\end{matrix}$

Where a′ is the apparent size of the object. a is the actual size of theobject. D is the object's distance from the camera, ϕ is the orientationangle. f is the focal length, and k is a conversion factor representingresolution, or the number of pixels per unit focal length. The object'sdistance can be expressed as a function of its average disparity value,d:

$\begin{matrix}{D = \frac{Bfr}{W_{p}\left( {d + C} \right)}} & {{Equation}\mspace{14mu}{A2}}\end{matrix}$

Where B is the baseline distance between the stereo-pair, r is the ratioof the camera's native resolution to the downsampled resolution used togenerate the disparity map. W_(p) is the physical width of a pixel, andC is an alignment coefficient that is given by:

$\begin{matrix}{C = {\frac{Bfr}{W_{p}D_{0}} - \frac{O_{H\; 0} - O_{H}}{2}}} & {{Equation}\mspace{14mu}{A3}}\end{matrix}$

Where D₀ is the distance to an object at zero disparity, defined as thebackground's disparity in this implementation, O_(H0) is the horizontalalignment setting, which yields zero disparity at D₀, and O_(H) is thecurrent alignment setting. As shown in figure A1, if the camera setup isoriented at an angle, ϕ, such that a portion of an object's size isprojected onto another axis, disparity (and in effect distance) must becorrected according to the change in the object's size. This can bepresented as:

$\begin{matrix}{D = \frac{Bfr}{W_{p}\left( {{d\;\sin\;\phi} + C} \right)}} & {{Equation}\mspace{14mu} A\; 4}\end{matrix}$

Combining Equations A1 and A4, an object's size in pixels can becomputed from its physical dimensions, a, and disparity value, d as:

$\begin{matrix}{a^{\prime} = {a\left( \frac{fk}{\frac{Bfr}{W_{p}\left( {{d\;\sin\;\phi} + C} \right)}} \right)}} & {{Equation}\mspace{14mu}{A5}}\end{matrix}$

Which can be rewritten as:

$\begin{matrix}{a^{\prime} = {a\left( \frac{k\;{W_{p}\left( {{d\;\sin\;\phi} + C} \right)}}{Br} \right)}} & {{Equation}\mspace{14mu}{A6}}\end{matrix}$

Similarly, a depth based scaling factor can be computed from the ratioof the size of an object at a neutral depth to the size of the object atthe current depth:

$s = {\frac{a_{neutral}^{\prime}}{a_{current}^{\prime}} = \frac{a\left( \frac{k\;{W_{p}\left( {d_{neutral} + C} \right)}}{Br} \right)}{a\left( \frac{k\;{W_{p}\left( {{d_{current}\sin\;\phi} + C} \right)}}{Br} \right)}}$

As a result, equation A6 can be simplified into:

$\begin{matrix}{s = \frac{d_{neutral} + C}{{d_{current}\sin\;\phi} + C}} & {{Equation}\mspace{14mu}{A7}}\end{matrix}$

If the resolution between the x and y dimensions is exceedinglydifferent, the scale can be adjusted to even the resolution, k, betweenthe dimensions. Assuming k_(x)>>k_(y), then equation A7 can besimplified and rewritten into its x- and y- components:

$\begin{matrix}{{s_{y} = {{\frac{k_{y}}{k_{y}}s} = s}}{s_{x} = {\frac{k_{y}}{k_{x}}s}}} & {{Equation}\mspace{14mu}{A8}}\end{matrix}$

Equation A8 governs the relationship between depth and scale for scalingbetween different camera configurations; however, the current disparitymust be re-mapped to the trained stereo pair according to real-worlddistance, where

$\begin{matrix}{{D = {\frac{B_{c}f_{c}\; r_{c}}{W_{p\; c}\left( {{d_{current}\sin\;\phi_{c}} + C_{c}} \right)}\mspace{14mu}{and}}}{d_{trained} = {\frac{1}{\sin\;\phi_{t}}\left( {\frac{B_{t}f_{t}r_{t}}{{DW}_{p\; t}} - C_{t}} \right)\mspace{14mu}{yield}\text{:}}}{d_{trained} = {\frac{1}{\sin\;\phi_{t}}\left( {\frac{B_{t}f_{t}r_{t}}{\frac{B_{c}f_{c}r_{c}}{W_{p\; c}\left( {{d_{current}\sin\;\phi_{c}} + C_{c}} \right)}W_{p\; t}} - C_{t}} \right)}}} & {{Equation}\mspace{14mu}{A9}}\end{matrix}$

In addition, the overall scale must be reduced by the reduced by theratio of the trained to the current focal length:

$S = {\left( \frac{f_{training}}{f_{current}} \right)\frac{d_{neutral} + \; C}{{d_{trainied}\mspace{11mu}\sin\;\varphi} + C}}$

Relationship with Other Electronics Devices

Given that such a device can interface with more than just a computer orlaptop, in accordance with one or more embodiments of the invention, theinventive keyboard device can be envisioned to also interface with a TV,SmartTV, tablet or a smartphone. As an example, the Asus™ Transformer,enables an Asus™ tablet with a keyboard. However, users are typicallyinclined to touch the screen, leaving marks and making the interactionmore cumbersome. Such an interaction can be simplified if sensors areintegrated into the keyboard portion to enable 3D gestures. Anotherpossible implementation involves interface a wireless keyboard with aSmart TV. The keyboard provides traditional computing functionality inthe living room, while at the same time enabling pointing towardsdifferent content on the screen to access information. Such a device issignificantly easier to use than a touch screen, an integrated tablet,or a mouse.

A new keyboard apparatus is therefore preferably presented thatintegrates a dual-camera system into the keyboard and enables 3Dgestures, while maintaining standard functionality of mouse, keyboard,and multitouch. Various variations of the apparatus are presented aswell as a representation of different hardware realizations. Interactionbetween the inventive keyboard, and one or more additional peripheralinput devices, such as a mouse and the like, may be exploited in orderto further determine desired gesture recognition timing and positioning.Finally, while the sensors in the keyboard have been described as stereocameras, other cameras, such as time of flight, or other camera systemsthat provide depth information may also be employed if desired.

It will thus be seen that the objects set forth above, among those madeapparent from the preceding description, are efficiently attained and,because certain changes may be made in carrying out the above method andin the construction(s) set forth without departing from the spirit andscope of the invention, it is intended that all matter contained in theabove description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

It is also to be understood that this description is intended to coverall of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall there between.

What is claimed:
 1. A gesture-enabled keyboard comprising: a keyboardhousing comprising one or more keyboard keys for typing; a camera sensormounted within the keyboard housing; a pair of mirror systems comprisinga plurality of individual mirrors for directing multiple views of ascene to the camera sensor; and one or more micro-electronic machines,each employed with a corresponding one or more of a plurality ofindividual mirrors for one or more of the pair of mirror systems inorder to adjust the view of the scene captured thereby.
 2. Thegesture-enabled keyboard of claim 1, wherein view of the scene from eachof the pair of mirror systems is directed to a different portion of thecamera sensor.
 3. The gesture-enabled keyboard of claim 1, wherein agesture region is defined within field of views of the pair of mirrorsystems.
 4. The gesture-enabled keyboard of claim 3, wherein the gestureregion further comprises a plurality of interaction zones.
 5. Thegesture-enabled keyboard of claim 4, wherein a virtual membrane defininga region of transition from one of the plurality of interaction zones toanother of the plurality of interaction zones.
 6. The gesture-enabledkeyboard of claim 5, movement of a gesture object from one of theplurality of interaction zones to another of the plurality ofinteraction zones through the virtual membrane defining the region oftransition therebetween defines a touch interaction.
 7. Thegesture-enabled keyboard of claim 6, wherein the gesture region isdefined as beginning a predetermined distance above the one or morekeyboard keys, and wherein gesture interaction is enabled when one ormore gesture objects are positioned within the gesture region, andwherein gesture interaction is disabled when no gesture objects arewithin the gesture region.
 8. The gesture-enabled keyboard of claim 7,wherein gesture interaction is enabled only when one or more gestureobjects are positioned within the gesture region, and one or morealternative input devices is not in use.
 9. The gesture-enabled keyboardof claim 8, wherein one or more alternative input devices comprise oneor more of a computer mouse, trackball, pointer and input pad.
 10. Thegesture-enabled keyboard of claim 8, wherein a background image in thefield of view of the camera sensor is updated when the one or morealternative input devices is in use, and the background image in thefield of view of the camera sensor provided by the pair of mirrorsystems is not updated when the one or more alternative input devices isnot in use.
 11. A method for inputting a command to a computing system,comprising the steps of: providing a keyboard further comprising akeyboard housing comprising one or more keyboard keys for typing; acamera sensor mounted within the keyboard housing; a pair of mirrorsystems for directing multiple views of a scene to the camera sensor,and one or more micro-electronic machines, each employed with acorresponding one of a plurality of individual mirrors for one or moreof the pair of mirror systems in order to adjust the view of the scenecaptured thereby, a field of view of the camera sensor directed by thepair of mirror systems projecting substantially perpendicularly to aplane of the keyboard housing; defining a gesture region within thefield of view of the camera sensor; determining whether one or moregesture objects are positioned within the gesture region; determiningwhether one or more alternative input devices are in use; and enablinggesture interaction when one or more gesture objects are positionedwithin the gesture region, and the one or more alternative input devicesare not in use.
 12. The method of claim 11, further comprising the stepsof: updating a background image in the field of view of the camerasensor when the one or more alternative input devices are in use; andfixing the background image in the field of view of the camera sensorwhen the one or more alternative input devices are not in use.
 13. Themethod of claim 12, further comprising the step of generating a depthmap of the one or more gesture objects in the gesture region whengesture recognition is enabled.
 14. A gesture-enabled keyboard,comprising: a keyboard housing comprising one or more keyboard keys fortyping; a camera sensor mounted within the keyboard housing; a pair ofmirror systems for directing multiple views of a scene to the camerasensor, each of the pair of mirror systems comprising a plurality ofindividual mirrors for directing a view of a scene to the sensor, a viewof the scene from each of the pair of mirror systems being directed to adifferent portion of the camera sensor, and wherein a field of view ofthe camera sensor projects substantially perpendicularly to a plane ofthe keyboard housing as directed by the pair of mirror systems, abackground of the field of view being updated when one or morealternative input devices are in use; and one or more micro-electronicmachines, each employed with a corresponding one or more of a pluralityof individual mirrors for one or more of the pair of mirror systems inorder to adjust the view of the scene captured thereby wherein a gestureregion comprising one or more interaction zones and a virtual membranedefining a region of transition from one of the one or more interactionzones to another of the one or more interaction zones is defined withinthe field of view of the camera sensor; wherein gesture interaction isenabled when one or more gesture objects are positioned within thegesture region, and when the one or more alternative input devices arenot in use.